Search Results

This article is based on a presentation given at the Journal of Vocational Education and Training 8th International Conference held in Worcester College Oxford, UK, in July 2009. The paper was written at the beginning of a wider research project, conducted within the Research Division. The aim of the wider project is to develop a research instrument for comparing the syllabus demands of cognate units from different types of qualifications. The specific aims of the present article are to review the theoretical approaches, methods and research instruments used to compare vocational qualifications (VQs) in England, with the view to gauging their appropriateness for comparing the demands of different types of qualifications. The wider project is still a work in progress.

Published abstract. Show less

Researchers were tasked with investigating methods of comparing general qualifications with vocational qualifications at level 2. Reading a variety of OCR (Oxford Cambridge and Royal Society of Arts Examinations) level 2 specifications illustrated that they included knowledge, skills and understanding from five domains; the affective, cognitive, interpersonal, metacognitive, and psychomotor domains. These domains were included in the instrument so it should be suitable for use with different types of qualifications. Comparability studies draw heavily on expert judgement and research about expert judgement has proved fruitful for other areas of assessment practice. Given that expert judgement research has a track record of being useful, it was decided to further investigate comparability judgement in the present study.

Edited excerpts from publication. Show less

Complex competences integrate a variety of skills. For example, models of professional proficiency or intelligent practice often incorporate the ability of a person to construct a holistic view of a problem or situation (Dreyfus & Dreyfus, 1980; Eraut, 1994). There is evidence that observational methods can be used to capture the integration of skills, knowledge and attitudes that pertain to higher level work (Eraut & Steadman, 1998). The aim of this research was to explore the measurement qualities of prevalent approaches to observation (checklists and Global Rating Scales [GRSs]) in the context of assessing complex competence.

Excerpt from publication. Show less

Historically, unitary awarding bodies and the national regulator monitored standards of qualifications between awarding bodies, over time and between cognate qualifications at the same level, and this work continues. A key reason for conducting such work is to avoid inequalities and inequities which would be created by the existence of easier routes to access further study or jobs. Ideally standards are compared in terms of candidates' performance and in terms of the demands of the qualifications. When comparing new qualifications there is sometimes a lack of performance evidence or assessment tasks to form a robust sample from which generalisable research results can be drawn. In such cases comparability studies could focus on specifications and the associated demands. However, studies restricted to one aspect of comparability (whether it be performance or demands) are limited. One approach to comparing demands of qualifications is for experts to rate them on a scale of cognitive demands known as [complexity, resources, abstractness, task strategy, response strategy] CRAS. CRAS was developed using academic qualifications. An issue deriving from its provenance may be that CRAS is not suitable for use with vocational qualifications which are different in nature and purpose to academic qualifications. Generally there are far more comparability studies about academic qualifications than [vocational qualifications/vocationally related qualifications] VQ/VRQs. In the present study [the authors] investigate whether CRAS is suitable for use in comparability studies which include VQs/VRQs.

Published abstract. Show less

There are thousands of diverse qualifications in the UK. Comparability studies about qualification standards generally use the following as comparators; quality of candidates' performance, and demand. For new and vocational qualifications, samples of candidates' performance and assessment tasks (e.g. examination questions) can be small or unrepresentative and thereby inappropriate for research purposes. Consequently, researchers employ other comparators including specification features, e.g. depth of knowledge. The article details the process of devising a research instrument to compare the features of cognate units from diverse qualifications and subjects. Such an instrument is atypical but valuable for comparability studies.

As part of a wider project about comparing different types of qualifications Kelly's repertory grid interviews elicited knowledge from twelve experts. They represented three subjects and composite, general, vocational and vocationally related qualifications. A secondary thematic analysis of the data was completed. The result was a series of features: learning; knowledge; summative assessment task; qualification system. Each feature had several sub-features. Both features and sub-features served to categorise the interview data. An instrument was derived from the features and sub-features, as well as the researchers' experience of qualifications. The instrument was refined through consultation with colleagues. The instrument in its final form consisted of a series of items relating to possible features of the different specifications. Respondents to the instrument were required to tick a box to indicate that the item applied to the given specification. A pilot of the instrument indicated that salient features vary somewhat between units. Therefore, as hoped, the research instrument highlighted the similarities and differences between units. This is the case for units of the same type and different types. However, there are no established conventions about how to analyse data. Therefore the instrument is suitable for use in future comparability studies about features, as long as the analysis of results is agreed from the outset. Future research might compare qualifications with data collected using the instrument.

Edited published abstract. Show less